DATA ENGINEERING / EVENT-DRIVEN STREAMING
AtmosFlow is an event-driven pipeline that streams live weather for 4 Vietnamese cities through Kafka into PostgreSQL, refines it with a dbt Bronze–Silver–Gold workflow on an Airflow schedule, and watches itself through Grafana.
01 / CONTEXT & PROBLEM
A weather API returns point-in-time JSON — no history, no consistency guarantees, no aggregates. The exercise: build the full production pattern (streaming ingestion, layered modeling, scheduled orchestration, observability) at a scale one person can run and debug, so every component's failure mode is understood rather than abstracted away.
02 / ARCHITECTURE
Weather API
4 Vietnamese cities
→
Python producer → Kafka
event stream · decoupled ingestion
→
PostgreSQL Bronze
raw landed events
→
dbt Silver → Gold
cleaned records → analytics-ready city summaries · full lineage
→
Grafana
pipeline & data monitoring
Airflow triggers the dbt refresh every 15 minutes · entire stack runs on Docker Compose
03 / KEY DECISIONS & RESULTS
Kafka between API and database
Overkill for 4 cities, intentional for learning: the producer/consumer split means ingestion hiccups never block the warehouse, and replaying a window is a consumer restart.
Bronze–Silver–Gold with full lineage
Every Gold-layer city summary traces back through documented dbt models to the raw event that produced it — the same medallion discipline used on enterprise lakehouses.
Observability from day one
Grafana dashboards watch both the infrastructure and the data freshness — a pipeline that can't tell you it's broken isn't done.
04 / WHAT I'D DO DIFFERENTLY
The pipeline trusts the API's schema; a silent field change would flow into Bronze before anything complained. Next: schema validation at the producer with a quarantine topic, plus dbt source freshness checks wired into the Grafana alerts.